Digital Assistant Extension Automatic Ranking and Selection

ABSTRACT

Representative embodiments disclose mechanisms to automatically rank and select extensions triggered in a digital assistant. A sample set of extensions are executed against a set of curated queries in order to extract a set of features and/or statistics. The system trains a machine learning model based on the features and/or statistics to rank and select extensions based on their response to a query. New extension incorporated into the system are executed against a second set of curated queries to obtain a set of extracted features and/or statistics which are saved for use at runtime. At runtime, a query phrase received by the system triggers one or more tasks from extensions. Extracted features for the triggered extensions are combined with stored features/statistics and at least a subset of the results presented to the trained ranking and selection model. The model ranks and selects appropriate tasks which are presented to the user.

FIELD

This application relates generally to extensions for digital assistants.More specifically, embodiments disclosed herein utilize machine learningto automatically rank and select digital assistant extensions.

BACKGROUND

Digital assistants have been making inroads in various aspects ofpeople's lives. The purpose of these digital assistants is to make thelife of the user easier by automating tasks, performing various tasks onbehalf of a user, and surfacing important information. The currentlyavailable digital assistants rely on a narrow set of information orcommands. Since digital assistants reside in closed environments, theowner of the digital assistant is responsible for evolving the narrowset of information or commands applicable to a particular digitalassistant.

It is within this context that the present embodiments arise.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example architecture of a digital assistant systemaccording to aspects of the disclosure.

FIG. 2 illustrates another example architecture of a digital assistantsystem according to aspects of the disclosure.

FIG. 3 illustrates an example architecture for a digital assistantsystem with automatic ranking and selection of bots.

FIG. 4 illustrates an example architecture for training a ranking andselection model.

FIG. 5 illustrates an example architecture for collecting information onsubmitted bots.

FIG. 6 illustrates an example of the various parts of an extension bot.

FIG. 7 illustrates an example flow diagram for automatically ranking andselecting bots to service a request.

FIG. 8 illustrates a representative machine architecture suitable forimplementing the systems and other aspects disclosed herein or forexecuting the methods disclosed herein.

DETAILED DESCRIPTION

The description that follows includes illustrative systems, methods,user interfaces, techniques, instruction sequences, and computingmachine program products that exemplify illustrative embodiments. In thefollowing description, for purposes of explanation, numerous specificdetails are set forth in order to provide an understanding of variousembodiments of the inventive subject matter. It will be evident,however, to those skilled in the art that embodiments of the inventivesubject matter may be practiced without these specific details. Ingeneral, well-known instruction instances, protocols, structures, andtechniques have not been shown in detail.

Overview

The whole purpose of digital assistants is to bring information to theattention of a user and to perform tasks on behalf of a user. In thecontext of this disclosure a digital assistant is a system, collectionof systems, service, collection of services, and/or combinations thereofthat collect information from or about a user, provide tips to the user,and/or perform various tasks on behalf of the user.

Thus, users want and expect digital assistants to provide a wide varietyof services and perform a wide variety of tasks. For example, a digitalassistant that is able to book a flight, make reservations, follow up onpackages, order items, pay bills, and a wide variety of other tasks isvaluable to a user. Each of these tasks tends to reside in a particulardomain. Rather than developing services and capabilities in every domainof interest to a user, manufacturers of digital assistants can open themup to third party extensions. These extensions can enrich and/or extendthe domains and tasks within a domain in order to improve thefunctioning and utility of a given digital assistant. Extensions areprovided in the form of executable instructions and/or data are used toextend the functionality of a digital assistant. In this disclosure,extension functionality is provided by “bots” which refers to theexecutable instructions and data (if any) provided by a developer andwhich is incorporated into the digital assistant. Conceptually, built infunctionality can also be considered to be provided by bots, although insome embodiments, the architecture of the built in functionality issomewhat different than a developer provided bot used to extend thefunctionality. However, the disclosed embodiments apply to both botsthat provide extension functionality and built-in functionality. Thus,unless otherwise indicated in a particular section, the term “bots” willbe used to refer to both extension bots and the executable instructionsthat provide built-in functionality.

A bot (extension or built-in) of the digital assistant may accomplishone or more “tasks” on behalf of the user. A single bot can accomplishone or more tasks. The tasks provided by a bot trigger on a set of“intents.” In this context an intent is something the user intends toaccomplish like “set a reminder” or “order pizza” and can be derivedfrom a query that the user submits like “remind me to schedule lunchwith Kisha next Thursday” or “can you order me some food?” or can bederived as explained below from one or more signals monitored by thesystem on behalf of the user. Intents reside in one or more domains,based on the type of event/activity they are intended to trigger, like areminder domain, an order domain, and so forth. Thus, a query submittedto a digital assistant is turned into one or more intents, which residein one or more domains. Language understanding models convert submittedqueries into one or more intents. In some embodiments, a developer hasthe option of using predefined intents or creating a new custom intentmodel based on sample queries and rules. In the context of thisdisclosure, the term “query” will be used to encompass any request,question, and so forth submitted by a user to a digital assistant inwhatever form.

Opening up a digital assistant to third party extensions brings with itits own set of problems. For example, where bots contain their ownintent models, it can be difficult or impossible to identify when a botwill be invoked (i.e., what set of queries will trigger the bot).Depending on the care and quality of the bot, the bot may suffer from avariety of issues such as over triggering (i.e., trigger when a userwould not want it to trigger), under trigger (i.e., not triggering whena user would want it to trigger), initiate an incorrectaction/mistakenly trigger (i.e., trigger an order food action when theintent of the user was to set a reminder to order food), have a “bad”user interactive experience, and so forth. Independent of any issuesthat may arise from the bot itself are other issues that can arisesimply by the existence of extension bots. For example, when multiplebots and/or built in functionality all want to trigger on the same userintent, how should the triggered bots be ranked? Should all the optionsbe presented to a user? Furthermore, the digital assistant system oftendoes not know the relationship between the extensions and the built infunctionality (i.e., whether a bot will be used to provide a differentfunctionality or the same functionality as what is built-in).

Embodiments of the present disclosure apply machine learning techniquesto automatically select and rank extensions and built in functionalityin order to fill a request (i.e., query) from a user. A model forranking and selecting triggered bots is created from applying machinelearning techniques to a representative set of bots and a set of curatedqueries.

The representative set of bots are selected from bots that exist in thesystem and/or bots that are created to mimic built-in functionality. Thebots are selected to provide a representative cross-section of domains,tasks, and so forth. The set of curated queries are created to identifyvarious aspects of the bots. The curated queries are run against the setof bots. The result is a set of features that describe various aspectsof the bots. The output is used to train a machine learning model thatis designed to rank and select one or more bots that are triggered byreceived queries. The curated query set can be quite large and isdesigned to ensure that the ranking and selection model is effectivelytrained. In addition, the features can be stored so they are availableto the system in a runtime situation.

When a new bot is to be added to the digital assistant, such as viasubmission through an extension portal, the new bot is run through a setof curated queries in order to gather the same features. The set ofcurated queries used in this process can be the same as, similar to, orcompletely different from, the set of curated queries used to train theranking and selection model. Typically, the set of curated queries willdiffer from the set of training curated queries at least in somefashion. The features are also stored in a data store for access duringruntime of the digital assistant.

During runtime, as a user submits a query, the digital assistant willapply its language understanding model to derive one or more intentsassociated with the query. Additionally, the query can be applied tobots that utilize their own intent model. The extensions and/or built-infunctionality that trigger tasks are evaluated and features are comparedto the features in the data store to identify the closest match. Thematched features are then passed to the ranking and selection modelpreviously created. The ranking and selection model ranks and selectsthose tasks (i.e., provided by some subset of the triggered bots) thatshould be presented to the user and the selected tasks are presented tothe user.

In this way, as new extensions are added to the digital assistantsystem, their behavior can be not only automatically integrated withbuilt-in functionality and existing extensions, but the behavior ofextensions can be modified (i.e., by the ranking and selection model) inorder to present a more seamless user experience.

DESCRIPTION

FIG. 1 illustrates an example architecture 100 of a digital assistantsystem 106 according to aspects of the disclosure. The figureillustrates how digital assistants 106 in the disclosure can be extendedby add on functionality. A developer submits a bot 114 providing one ormore tasks to a digital assistant extension portal 112, typically over anetwork 116. The digital assistant extension portal 112 takes in the bot114 and performs tests and any other functions that are required beforeapproval to deploy the bot 114 is received. These can include a widevariety of tests to ensure that the bot functions properly (i.e., noexecution errors) as well as security checks, and so forth.

As discussed below, one aspect in accordance with the present disclosureis to collect features to see how the submitted bot responds to acollection of curated queries and to collect description informationabout the bot. The description information allows the bot to beintegrated into the digital assistant. The features allow the system tounderstand how the bot fits in with the rest of the functionalityalready in the digital assistant (whether natively or throughextensions) and to identify what sort of intents the bot triggers on. Italso allows the bot to be tested for over triggering, under triggering,mis-triggering and other such problems discussed above. It furtherallows identification of how the bot performs in representative“dialogs” with a user and determine whether the bot appropriatelyresponds to follow on queries. Thus, a representative dialog might be:

User: “I'm hungry”

Assistant: “would you like me to order you some food?”

User: “remind me to order food at 12:30”

Assistant “I'll remind you.”

In this dialog the Assistant correctly interprets a need the user has,such as an intent to eat, and correctly interprets the user's furtherspecified intent not to eat now, but to be reminded at a later time.Thus, in addition to a single query aspect (i.e., a query that triggersan indicated response by the bot), the set of curated queries cancontain a series of queries to profile how a bot responds to follow onquery dialogs. In effect, running the set of curated queries against thebot prior to deployment allows the system to build a profile of thebehavior of the bot and to gather characteristics of the bot prior todeployment and thus be able to correctly prioritize the bot's responseagainst other bots that provide the same or similar tasks.

Once the system performs any desired tests and collects the descriptioninformation and the features, the bot 114 is deployed to the digitalassistant 106 where it provides extension action(s) 110 along-side thebuilt in native actions 108.

A user 102 submits one or more queries to the digital assistant 106,generally over a network 104 (which could also be the same network asnetwork 116). The digital assistant 106 determines one or more userintents associated with the submitted query and provides an appropriateresponse to the query. This process is discussed in greater detailbelow.

FIG. 2 illustrates another example architecture 200 of a digitalassistant system according to aspects of the disclosure. Such anarchitecture shows one possible embodiment where the various aspects ofa digital assistant system, such as digital assistant system 106, can beimplemented in a three-tiered architecture, comprising a front-endlayer, an application logic layer and a data layer. Such an architectureis only one type of cloud and/or server architecture which would besuitable for the disclosed embodiments and numerous other models andarchitectures could also be used. For example, more recently manyservices and applications are developed using a microservicearchitecture so that they can be deployed and executed on clusters ofsystems and/or service fabrics. However, the three-tiered architectureis sufficient for explanation purposes and those of skill in the artwould be able to understand how to implement the digital assistantarchitecture in other environments from the explanation provided.

As is understood by skilled artisans in the relevant computer andinternet-related arts, each module or engine shown in FIG. 2 representsa set of executable software instructions and the corresponding hardware(e.g., memory and processor) for executing the instructions. To avoidobscuring the disclosed subject matter with unnecessary detail, variousfunctional modules and engines have been omitted from FIG. 2.

As shown in FIG. 2, the front end layer comprises a presentation engine212 which can include one or more software and/or hardware implementedmodules. The presentation engine is responsible for the user experience(UX) and receives information and/or requests from variousclient-computing devices including one or more user device(s) 204, andcommunicates appropriate responses to the requesting device. Thus, thepresentation engine 212 can cause presentation of information (e.g.,cards, responses, etc.), user interfaces, and so forth on the userdevices 204, generally through the client application(s) 206.

In this embodiment, user devices 204 can include various devices caninclude devices such as a wearable device (i.e., watch, band, glasses,and so forth), a carry-able device (i.e., mobile phone, tablet, laptop,and so forth), a stationary or semi-stationary device (i.e., portablecomputer, desktop computer, hub, and so forth), and/or server devicesand/or services (i.e., servers, cloud services, calendar systems, emailsystems, and so forth). In addition, one more vehicles can be considereduser devices in the context of the embodiment of FIG. 2, even thoughsuch vehicle(s) may be associated with different users at differenttimes, such as where a vehicle is shared by more than one user.

A user interacts with all of these devices and/or services and they allhave information that can be used by the digital assistant system 102 toprovide digital assistant services to the user. This data will becollectively referred to as user data and can include informationrelated to the user as well as information about the user such aspreferences, profile information and so forth. Example informationincludes, but is not limited to, a user's calendar and/or schedule, todo list, email history, purchase history, a user's normal routine, routeto work, home location, work location, school location (i.e., for astudent), preferred forms of communication, devices owned/interactedwith, and so forth.

The user device(s) 204 may execute different client applications 206that allow the device to communicate with the digital assistant system106, typically through the presentation engine 212. Such applicationscan include, but are not limited to web browser applications, a digitalassistant application, and/or client applications (also referred to as“apps”) that have been developed for one or more platforms to providefunctionality to present information to the user and communicate via thenetwork(s) 112 to exchange information with the digital assistant system106. Each of the user devices 204 may comprise a computing device thatincludes at least a processor, a display and communication capabilitieswith the network 112 to access the digital assistant system 106. One ormore users 202 may be a person, a machine, or other means of interactingwith the user device(s) 204. The user(s) 202 may interact with thedigital assistant system 106 via the user device(s) 204. The user(s) 202may not be part of the networked environment, but may be associated withuser device(s) 204.

In accordance with privacy laws and with user authorization, the digitalassistant system 106 can store and/or have access to a wide variety ofdata. As shown in FIG. 2, the data layer includes several databasesand/or other types of data sores (collectively data stores), including adata store 218 for storing data associated with a user, such as user202. The types of information that can be stored in such a data store,includes, but is not limited to, any user information discussed above.

The data layer can also include a third party data store, which storesinformation obtained from third parties, such as from third partysystem(s) 208 and/or third party application(s) 210. This informationcan be about the user, or may be information used by service actions toprovide the service utilized with the service action. For example, ifthe service action further utilizes services provided by the third partysystem(s) 208 and/or third party application(s) 210, informationallowing the interoperation between the digital assistant system 106 andthe third party system(s) 208 and/or third party application(s) 210 maybe stored in third party data store 220. Information gathered from othersources may also be stored in third party data store 220 in someembodiments.

Another data store in the data layer can contain rules 222 for theexecution engine 216, the language understanding model 224, and/or othermodules, components, bots, etc. to operate on. Such rules specify a set(combination) of input data, a set of conditions and a set of resultantoutput, such as user intents or actions that should be taken. In thiscontext a set can consist of one or more items (i.e., one or more inputdata, one or more conditions, and/or one or more resultant outputs). Inother words, a rule defines a relationship between input data andresultant output(s). An example rule might be that when a user is withina threshold time from an appointment, then send a card to the userreminding the user to leave for the appointment. This example ruledefines a relationship between user information (the user with theappointment, the appointment), and a resultant action (send a reminderto leave for the appointment). Conditions can be implied, derived, orexpress, such as when the time within a designated threshold. Anotherexample rule might be when a user input query contains some form of theterm “reminder” the reminder intent should be set.

The digital assistant system 106 also comprises data intake and/orenrichment modules 214. As data is received, the data can be processedin a variety of ways, including any combination of pre-processingoperations such as validation, cleaning, normalization, transformation,feature extraction, selection, and so forth. Validation and/or cleaninggenerally includes of detecting and correcting data or removing datathat is corrupt, inaccurate, incomplete, etc. Validation also ensuresthat data is of the proper type. Normalization generally includesensuring data is scaled appropriately, redundancies are removed, and soforth. Transformation includes mapping data from one set to another,such as when data from various sources are mapped to an equivalent dataset (i.e., where part numbers from different manufacturers aretransformed with a standard part numbering system) and so forth. Featureextraction includes identifying and extracting various pieces ofrelevant information is sometimes related to machine learning. Featureextraction can also involve reducing the dimensionality of a data set.Selection includes selecting a subset of relevant features that are ofinterest. Other pre-processing may also be used as appropriate.

The data intake and/or enrichment module 214 may also perform dataenrichment. Data enrichment often correlates information from differentsources and assembles it in a way that the data becomes enriched with(i.e., related to) other information. For example, user information frommultiple sources can be assembled in one or more associated datastructures. The assembled information can then allow the digitalassistant system to identify where a user is, what activities the useris engaged in, and so forth.

The data intake and/or enrichment module 214 can receive and assembleinformation from a variety of source, represented in FIG. 2 by the thirdparty system(s) 208, the third party application(s) 210, the userdevice(s) 204 and other sources. By way of example only, the third partyapplication(s) 210 and/or the third party system(s) 208 can representservices like email, scheduling/calendaring, and other such services.The third party application(s) 210 and/or the third party system(s) 208can also be systems used by bots that provide extension services to filluser requests. For example, a bot may utilize a flight reservationsystem when the user requests the assistant to book a flight to adestination on a particular date. The third party application(s) 210and/or the third party system(s) 208 can also represent other inferencesystems like hubs which assemble information and make inferences. Thesecan infer things like home/work/school location and activities, travelcontext, location and visiting patterns, route patterns, meetingcontexts, communication contexts, emergency information, search contextsand so forth. Of course given the raw data, the digital assistant system106 can also make such inferences in some embodiments.

The digital assistant system 106 comprises a language understandingmodel 224. The language understand model 224 is responsible for takingan input query and for identifying one or more intents from the inputquery. The language understanding model can associate a confidence level(also referred to as a confidence score) with the identified intents.The confidence level represents the confidence that the input query mapsto the identified intent. Prior to being passed to the languageunderstanding model 224, input queries can be placed into a formatsuitable for processing by the language understanding model 224. Forexample, queries received in audio format (voice, etc.) can be convertedto text or some other desired format prior to being passed to thelanguage understanding model 224.

The digital assistant system 106 also comprises an execution engine 216.The execution engine 216 receives intents from the languageunderstanding model 224 and/or from other executable instructions andidentifies which bots are triggered by the intent. The bot can triggernot only on the intent, but the intent/confidence level combination. Insome embodiments, a bot may not trigger if a given intent has aconfidence level below a threshold, but may trigger above thatthreshold. In other embodiments, confidence level is not used and theresponse selection and ranking model appropriately handles anyambiguity.

The query/response model represents operation of the digital assistantsystem 106 when the system is acting in a reactive mode. However, thedigital assistant system 106 can also operate in a proactive mode. Inthe proactive mode, the system can monitor the data (current context ofthe user and/or other signals or events) and proactively take action(i.e., engage the user in some fashion) without the user initiating theaction, such as through a query. For example, by accessing the user'scalendar, travel schedule and/or flight information, the digitalassistant system 106 may know that the time to check-in for the user'sflight has arrived. The digital assistant can proactively engage theuser with a reminder (i.e., time to check in), with a request (i.e., itis time to check in, should I do that for you?), or in some other way.In yet another example, if the user is browsing restaurants, the digitalassistant 106 my step in and ask whether the user would likerecommendations for a restaurant, make reservations, and so forth.

The execution engine can also monitor incoming data and provide insightsas part of the trigger identification process. Insights, as used herein,are a specific piece of information about a user that the system haspermission to access and/or deduce about a user or set of users (i.e.,from data about or related to the user). Insights can be representativeof a particular context and/or activity of the user such as in ameeting, operating a vehicle, browsing the internet, working, at astore, on vacation, and so forth.

The insight(s) and/or intent(s) trigger bots/built-in functionality ofthe digital assistant 106. These are represented, for example, in FIG. 2by the native tasks 266 and the extension tasks 230. Once thebots/built-in functionality are triggered, the task selection operation228 ranks and selects the tasks and finalizes the response to the user,as described below.

FIG. 3 illustrates an example architecture 300 for a digital assistantsystem with automatic ranking and selection of bots. During runtime, auser will use a device 302 to submit a query to the digital assistantsystem, typically via a network 304 as previously explained. The querywill be converted to a format desired by the language understandingmodel 308. For example, if the query is audio (voice, etc.), the querycan be converted by a speech recognizer 306 that performs speech to textconversion and places the text into the format desired by the languageunderstanding model 308.

The language understanding model 308 produces one or more insightscorresponding to the query. These insights are passed along to theexecution engine 310. As previously discussed, execution engine 310 canalso provide insights as part identifying which bots are triggered bythe query and/or insights. However, for ease of description and to avoidrepetition of the phrase “queries and/or insights” or the phrase“intents and/or insights,” the description will be presented in terms ofa bot being triggered by a query. However, in all embodiments describedherein, the trigger by query/intent also includes triggering by intentas previously described.

The execution engine 310 identifies the bots that are triggered by thequery. Additionally, the query can be passed directly to those bots thatuse their own language understanding model. The execution engine 310identifies bots triggered by the query by identifying those bots thattrigger on a particular insight/query combination and/or by presentingthe query to those bots that use their own language understanding model.Bots are triggered when they indicate that they want to handle thequery.

In identifying the bots that are triggered, the execution engine canmake use of definitions and other information in data store 322. Thedefinitions in data store 322 help define the various tasks andexperience provided by the bots. Thus, the data store 322 holds the listof experiences (i.e., bots and/or domains), task descriptions, and soforth used by the execution engine 310.

In addition, the execution engine 310 extracts features from and/orabout the triggered bots. The execution engine 310 extracts the featuresrelated to the following:

-   -   features computed looking at how one specific bot responded to        the specific query, including the language understanding model        outputs;    -   features computed looking at how all the activated bots        responded to the specific query, including the language        understanding model outputs;    -   features computed looking at the statistics of one specific bot;        and    -   features computed looking at the statistics of all the activated        bots.

Representative statistics from which features are computed are (i.e., inthe last two bullets above) are discussed below. Additionally, in someembodiments, a model may be used to convert the output of a bot into“normalized” data before the statistics and/or features are extracted.The above features and examples thereof are discussed below.

The feature extractor 312 identifies features that have beenpre-calculated (as explained below) and that are associated with thetriggered bots. As discussed below, features are calculated when a botis presented for deployment and these features are used, in conjunctionwith the features above, to identify how the bot should be ranked andwhether the bot should be selected. The feature extractor 312 retrievesthe features associated with the individual triggered bots from datastore 322, combines the retrieved features with the features fromexecution engine 310 and presents the features and/or subset thereof tothe response ranking and selection model 314. In some embodiments aclosest matching set (i.e., using a distance measure) is retrieved fromdata store 322. In this disclosure a subset includes a proper subset(i.e., the complete set). This model is created through a machinelearning process as described below and maps the presented features to aranking for the triggered bot.

The output of the model is a ranked list of the triggered bots. Theranking assigned to a triggered bot can be an absolute or relativeranking. Absolute rankings effectively place the ranked bots into“buckets” that represent different treatment of the triggered bots. Forexample, one bucket can signal to the selection logic that the triggeredbot would be suitable for presentation to the user (i.e., that the bothas appropriately triggered and would be appropriate to present to theuser in response to the query) while another bucket can signal that thetriggered bot would not be suitable for presentation to the user inresponse to the query for some reason. Relative rankings place thetriggered bots into a list by order of some metric, such as confidencelevel that the triggered bot would meet with acceptance by the user(i.e., triggered as it was supposed to and provides a task appropriateto the query), whether the task provided by the bot is favored by theuser (i.e., based on a user profile or other data), or some othermetric.

Additional selection logic can be applied to select a subset of theranked list to present to the user. The selection logic can account forvarious aspects, such as user profile data, a maximum number of choicesthat should be presented to the user, some cost metric, or othercriteria. As a representative example, suppose the request was to book aflight. Triggered bots include bot A that uses a particular bookingservice and bot B that uses another booking service. The featureextractor 312 identifies the features associated with the two bots andpresent a subset to the ranking and selection model 314. Suppose theranking and selection model 314 ranks them as bot B and then bot A in arelative ranking list.

In a first instance, the user has not indicated a preference for abooking service and the maximum number of choices that the system willpresent to a user is 3. In this instance, the response back to the userwill as the user to pick a booking service to use. Depending on theuser's choice, the selected bot will be invoked and the ticket booked.

In a second instance, the user has indicated a preference for thebooking service used by bot A. In this instance the response back to theuser may indicate that the ticket will be booked using the bookingservice associated with bot A and the system will invoke bot A to bookthe service.

In both these instances, additional selection logic can be appliedeither to select the bot(s) will be used to finalize the response. Thiscan be performed by the response ranking and selection model 314 and/orthe response finalization operation 316. Once the bot(s) have beenselected the response finalization operation 316 prepares theinformation and presents it to the user in an appropriate format, suchas previously described.

As should be evident by this discussion, the ranking and selection model314 has the effect of not only identifying the bots that should providethe tasks to the user, but also has the effect of modifying the behaviorof the bots. Thus, if a bot triggers when it should not, as the featuresfor that bot are applied to the ranking and response model, the modelwill rank the bot in a manner or otherwise indicate that it should nothave triggered and should not be presented as part of the response. Thishas the effect of modifying the behavior of the bot and of bringingunruly bots in line with how they should operate in an automatic way.

As discussed more completely in conjunction with FIG. 5 below, bots 318used to extend the functionality of the digital assistant are submittedvia an extension portal 320 which applies any acceptance/approvalprocess (testing for errors, etc.) as well as using a set of curatedqueries to gather the features (stored in the data store 322) discussedabove that are used by the response ranking and selection model 314 torank and select the triggered bots as previously described.

As discussed more completely in conjunction with FIG. 4 below, a sampleset of bots 326 and a curated set of queries 328 are used by a machinelearning method 324 to produce the ranking and selection model used byresponse ranking and selection 314.

FIG. 4 illustrates an example architecture 400 for training a rankingand selection model. As previously described a digital assistant system402 comprises an execution engine 404, a feature extractor 406 and aranking and selection model 408. The functions of these blocks have beenpreviously described and there is no need to reiterate what these blocksdo in the runtime digital assistant 402.

The annotation store 412 contains a representative set of bots 416 and aset of curated queries 418 that are used to train the ranking andselection model 408 as explained below. The bots and queries extractionfunction 414 retrieves the representative set of bots 416 and thecurated queries 418 from the annotation store 412. Curated queries, alsosometimes referred to as annotated queries, are queries chosen to gatherthe features previously referred to. The details of representativefeatures are presented below. The queries are curated in the sense thatthey are selected to gather the feature information needed to build a“profile” on how a particular bot operates. Queries can include singlequeries such as a query that should trigger an action all by itself(“schedule an appointment with my doctor for Friday at 2 pm”) as well asa set of queries and follow on queries that together represent aconversation under the hypothesis that a specific response (i.e., by thebot) is accepted and presented to the user. An example might be theexchange:

-   -   User: “Can you schedule me a doctor appointment?”    -   Assistant: “This would be your primary care physician: Dr.        Wallenberg?”    -   User: “Yes.”    -   Assistant: “When would you like the appointment scheduled?”    -   User: “Friday at 2 pm”    -   Assistant: “Your appointment has been scheduled and placed on        your calendar.”

For bots that use customized language understanding models (instead ofthe built in LU model), the queries can contain a variety of queriesused to gather statistics on how the LU model behaves and/or what typeof queries trigger the bot. These statistics become part of the featuresthat are used to train the ranking and selection model.

Operation 420 extracts descriptions (also referred to as definitions)that describe the bots. The descriptions are stored in a data store, forexample in data store 410, in some embodiments. The data store 410 maybe the same data store as the data store 322 of FIG. 3. The descriptionsallow the data store 410 to contain the list of tasks, theirdescriptions and so forth that are used by the execution engine 404during operation to identify the bots triggered by a query/intent(s).

Operation 422 extracts the features discussed herein that are used totrain the ranking and selection model. The features may include, but arenot limited to, one or more of:

-   -   1. Features extracted from the language understanding results        (for each single possible response in the curated query set and        for all possible responses altogether in the curated query set).        Examples include but are not limited to: the minimum, maximum,        average, sum of the language understanding model confidence        scores, the number of different domains identified, the number        of intents identified, the number of entities detected across        all choices, and/or the number of entities detected for a bot.    -   2. Features extracted from how well a conversation progress        under the hypothesis that a specific response is accepted and        presented to the user (whether the bot triggers when it should        and whether the response to the query was appropriate as        determined from the curated query set). Examples include but are        not limited to: the number of new entities filled, the number of        entities revisited, whether confirmation is asked, whether the        task triggered was a repeat question action, whether the task        was triggered a new question action, and/or the dialog state        confidence increase.    -   3. Features extracted from the statistics collected for the        various models used by the various tasks activities of        this/these bot(s). Examples include, but are not limited to:        predicted general precision, predicted general recall, expected        precision for the current language understanding model result,        and/or estimated semantic confidence rescaling factor.    -   4. Features extracted from the bot configuration. Examples        include, but are not limited to: number of tasks the bot        performs, number of parameters in the task, type of LU models        used, type of queries used, etc.    -   5. Features extracted from the input source:        speech-recognition-level features for voice, textual        characteristics features for text queries, other-modality        specific information in the case other modalities are used        (gesture, ink, button press, etc.).

The statistics, like the other features above, are gathered by runningthe curated query set against the sample set of bots. The statistics mayalso comprise one or more of:

-   -   How often a bot is triggered by its own language use model (if        it uses one);    -   Which intents/categories of intents trigger a bot;    -   The average confidence score of the queries that triggered the        bot;    -   What is the confidence score distribution and/or probability        density function;    -   Characteristics of the query that triggered the bot (long query,        short query, etc.); and    -   How often does the model trigger on top of an existing intent we        have in the system.

Operation 422 extracts the features by executing the curated query set418 against the sample bots 416 and collecting the statistics and otherinformation identified above. For some of the information above, thequeries need not be executed and the information comes directly fromeither the queries 418 themselves, the bots 416, or both. For example,features extracted from the input source (i.e., #5) can be identified byeither examining the curated queries 418 or by evaluating metadata(annotations, etc.) associated with the curated queries 418. Featuresextracted from the bot configuration (i.e., #4) can be identified byexamining the bot configuration information and/or metadata associatedwith the bots. The extracted features can be placed in a series offeature vectors and stored in a data store 410 for use during runtime aspreviously explained. The data store 410 can be the same as data store322 of FIG. 3.

The series of feature vectors are used to train the ranking andselection model via operation 424. In operation 424 a machine learningmodel is used on the input feature vectors to train the ranking andselection model. Any number of machine learning models are suitable foruse in operation 424 and such models are well known. As representativeexamples only, the machine learning model can be one or more of aBayesian model (Native Bayes, Basian Belief Network, Basian Network,Averaged One-Dependence Estimators, etc.), a decision tree model(classification and regression tree, chi-squard automatic interactiondetection, conditional tree, etc.), a dimensionality reduction model(Principal Component Analysis, Partial Least Squares Regression,Projection Pursuit, Principle Component Regression, a discriminantanalysis model such as quadratic, regularized, flexible, linear,Multidimensional Scaling, etc.), an instance based model (k-nearestneighbor, learning vector quantization, self-organizing map, locallyweighted learning, etc.), a clustering method (k-means, k-median,expectation maximization, hierarchical clustering, etc.), deep learningmodel (deep Boltzmann Machine, deep belief networks, convolutionalneural network, etc.), an ensemble model (random forest, gradientboosting machines, bootstrapped aggregation, etc.), a neural networkmodel (radial basis function network, perception, back propagation,Hopfield network, etc.), a regularization model (ridge regression,elastic net, least angle regression, etc.), a rule system (cubist, onerule, zero rule, repeated incremental pruning to produce errorreduction, etc.), a regression model (linear regression, least squaresregression, logistic regression, etc.) or a model that is not classifiedin the above taxonomy such as support vector machine (SVM) or anothermodel.

The process is used to bootstrap the learning model. Once the initialmodel is trained, as additional bots are submitted and additionalstatistics extracted as described below, the statistics can be used toadjust operation of the raking and selection model 408 either on acontinuous or periodic basis. Thus, the statistics collected through botsubmission can be applied to a machine learning technique to furthertrain the model. In alternative embodiments, the model is trained on therepresentative set of bots 416 and the set of curated queries 418 and isnot updated. Additionally, or alternatively, an updated set ofrepresentative bots 416 and/or an updated set of curated queries 418 canbe used to update the training of the model.

The ranking and selection model is used by the ranking and selectioncomponent 408 of the digital assistant system 402 as described above,such as in FIG. 3.

FIG. 5 illustrates an example architecture 500 for collectinginformation on submitted extension bots 502. In other words, the examplearchitecture 500 describes the processes used by an extension portal 504(e.g., extension portal 320 or digital assistant extension portal 112)to collect information on submitted bots so they can be integrated intothe system and be ranked/selected by the ranking and selection model.The portal may perform additional processes not shown or discussed heresuch as security checks, and any other checks that happen beforedeployment is made.

Bot 502 is submitted through the extension portal 504. When bot 502 issubmitted, the developer defines the various intents and otherinformation (e.g., insights) needed to perform their tasks. Aspreviously discussed, the developer can rely on the built in LU modeland the intents created thereby or can create a new custom LU model. Insome embodiments, the custom model is created by giving example queriesand/or rules. The queries/rules can be based on positive examples (i.e.,trigger the bot when the query “book a flight” is received) or negativeexamples (i.e., do not trigger the bot when the query “book a flight” isreceived) or a combination of both.

When the bot is submitted, the extension portal 504 extracts descriptioninformation 506 and features 508. This is accomplished by running a setof curated queries 510 against the bot. The set of curated queries 510can be the same as, completely different from, or partially differentfrom the set of curated queries 418 used to train the ranking andselection model as described in conjunction with FIG. 4 above. Thedescription is extracted in the same fashion as the descriptions for therepresentative set of bots 416 above. In other words, the same operation420 can be used by extension portal 504 to extract the appropriatedescription. Similarly, the features 508 are extracted in the samefashion as described above in conjunction with operation 422, with theexception that the curated query set 510 is used to identify thefeatures.

The extracted features 508 can be used in some embodiments to update theranking and selection model in accordance with the machine learningmodel used to create the ranking and selection model. As previouslydescribed, this can be done on a periodic or aperiodic basis or not atall, if the embodiment doesn't use the extracted features to update themodel.

The description 506 and features 508 are stored in a data store 512.Data store 512 can be the same data store 322 as that illustrated inFIG. 3. The stored information is then used by the digital assistantsystem 514 during runtime as previously described.

FIG. 6 illustrates an example of the various parts of an extension bot602. The parts illustrated and discussed can be received and deployed aspart of a unitary data structure or may be received and deployed in datastructures that combine one or more of the illustrated parts.Furthermore, not all parts may be included in all bots. Bots can haveone or more of the parts described below in any combination.

An extension bot 602 can comprise user data 604 that the bot needs inorder to operate. For example, if a task the bot provides is booking aflight, the bot will need the user's name and other information requiredby the booking company to book the flight. Some of this information isuser data that the digital assistant has access to and that the botneeds to be given access to (with the user's permission) in order toaccomplish its task. Such user data can, for example, be obtained fromthe user and/or from a data store such as user profile and data store218.

As previously discussed, an insight is a specific piece of informationabout a user that the digital assistant has access to or can deduce fromthe information that the digital assistant has access to. Insights areoften related to context information, such as the user is in a meeting,driving a vehicle, at home, and so forth. Insights 606 represent theinsights that the bot 602 wants access to in order to do the work. Forexample, if the user says “order me a pizza,” the bot needs to knowwhere to deliver the pizza, which is presumably the user's currentlocation in the absence of any other direction from the user. Thus, thebot may need access to the user's current location insight in order tofill its task.

Intents 608 represent the intents that the bot would like to “subscribe”to. In other words, these are the intents produced by the built in LUmodel that should trigger the bot. In addition, the intents can have anassociated confidence score to allow the bot to decide whether there issufficient confidence in the intent to trigger and/or confidence scorethreshold that the digital assistant should trigger the bot on.

As previously discussed, in some embodiments a bot may have its owncustom LU model in order to determine its own intents. The custom LUmodel can be described by positive and/or negative example queriesand/or rules. Language understanding part 610 represents the custom LUmodel and/or, for those embodiments that use queries and/or rules todescribe the LU model, the queries/rules used to describe it.

Request/pre-service part 612 represents executable instructions thatshould be called prior to calling the actual instructions that providesthe task. The executable instructions can accomplish any pre-processingneeded and/or desired such as asking the user for information needed tocomplete the task if it is otherwise unavailable.

Processing/providing service part 614 represents executable instructionsthat accomplish the task the bot performs in accordance with the insightthat triggered the task.

Notification/post service part 616 represent executable instructionsthat should be called after the main part 614. These instructions canaccomplish any post-processing tasks that are needed such as data cleanup, task termination, etc.

The bot 602 can perform more than one task. For those instances, theparts above may have some parts that are common to all tasks, some partsthat are unique to a task or a combination thereof. In other words, insome instances, each task may have completely separate parts. In otherinstances, some parts may be common between multiple tasks.

FIG. 7 illustrates an example flow diagram 700 for automatically rankingand selecting bots to service a request. The flow diagram begins atoperation 702 and proceeds to operation 704 where the query is receivedby the digital assistant system. As previously discussed, the query cancome in any form such as audio (e.g., voice), text, ink, button press,and so forth.

Optional operation 706 represents any conversion done on the query toconvert it to a desired format. The query is presented to the LU modelin operation 707 and one or more intents are identified from the queryas previously described. The identified intents can also have anassociated confidence level that identifies the confidence that theintent is associated with the query.

Operation 708 identifies which bots trigger on the identified intent(s)and/or based on the query that was presented. This can be accomplishedas previously described above.

Operations 710-720 represent a loop that is performed for each bot thathas been triggered in operation 708. The loop begins at operation 710and proceeds to operation 714 where the features associated with thetriggered bot are identified. As previously discussed this can beaccomplished in a variety of ways, such as by matching features in datastore (e.g., data store 322, etc.) with a triggered bot, such as byusing an identifier, by using information extracted from the bot duringthe triggering process and so forth.

Operation 716 presents the features retrieved from the data store to theranking and selection model and obtains a rank for the bot. Aspreviously described, the rank can be relative, absolute, or somecombination thereof. The bot is placed in rank order (operation 718).

Operation 720 represents the end of the loop. After the loop isfinished, the system has a ranked list of the activated bots.

Operation 722 selects the bots in accordance with selection criteria. Asdiscussed, the selection criteria can comprise one or more of: selectionbased on a threshold (i.e. the top n ranked bots), selection based onuser profile data, selection based on the type of bot (i.e., built invs. extension), selection based on the category of the bot (i.e.,performs tasks in a given domain), and/or so forth.

Operation 724 presents the selected bots to the user in one or moreresponses. The response can be appropriate for the channel best suitedto communicate with the user. For example, if the user is operating avehicle, the response could be presented in audio fashion rather than bytext. This, operation 724 represents the functionality provided, forexample, by operation 316 of FIG. 3 and includes any combination offormatting the response for a given response medium (audio, text,tactile, etc. and combinations thereof) and a response channel (mobilephone, desktop screen, band/watch or other wearable, etc. andcombinations thereof).

The method ends at 726.

Example Machine Architecture and Machine-Readable Medium

FIG. 8 illustrates a representative machine architecture suitable forimplementing the systems and so forth or for executing the methodsdisclosed herein. The machine of FIG. 8 is shown as a standalone device,which is suitable for implementation of the concepts above. For theserver aspects described above a plurality of such machines operating ina data center, part of a cloud architecture, and so forth can be used.In server aspects, not all of the illustrated functions and devices areutilized. For example, while a system, device, etc. that a user uses tointeract with a server and/or the cloud architectures may have a screen,a touch screen input, etc., servers often do not have screens, touchscreens, cameras and so forth and typically interact with users throughconnected systems that have appropriate input and output aspects.Therefore, the architecture below should be taken as encompassingmultiple types of devices and machines and various aspects may or maynot exist in any particular device or machine depending on its formfactor and purpose (for example, servers rarely have cameras, whilewearables rarely comprise magnetic disks). However, the exampleexplanation of FIG. 8 is suitable to allow those of skill in the art todetermine how to implement the embodiments previously described with anappropriate combination of hardware and software, with appropriatemodification to the illustrated embodiment to the particular device,machine, etc. used.

While only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example of the machine 800 includes at least one processor 802(e.g., a central processing unit (CPU), a graphics processing unit(GPU), advanced processing unit (APU), or combinations thereof), one ormore memories such as a main memory 804, a static memory 806, or othertypes of memory, which communicate with each other via link 808. Link808 may be a bus or other type of connection channel. The machine 800may include further optional aspects such as a graphics display unit 810comprising any type of display. The machine 800 may also include otheroptional aspects such as an alphanumeric input device 812 (e.g., akeyboard, touch screen, and so forth), a user interface (UI) navigationdevice 814 (e.g., a mouse, trackball, touch device, and so forth), astorage unit 816 (e.g., disk drive or other storage device(s)), a signalgeneration device 818 (e.g., a speaker), sensor(s) 821 (e.g., globalpositioning sensor, accelerometer(s), microphone(s), camera(s), and soforth), output controller 828 (e.g., wired or wireless connection toconnect and/or communicate with one or more other devices such as auniversal serial bus (USB), near field communication (NFC), infrared(IR), serial/parallel bus, etc.), and a network interface device 820(e.g., wired and/or wireless) to connect to and/or communicate over oneor more networks 826.

Executable Instructions and Machine-Readable Medium

The various memories (i.e., 804, 806, and/or memory of the processor(s)802) and/or storage unit 816 may store one or more sets of instructionsand data structures (e.g., software) 824 embodying or utilized by anyone or more of the methodologies or functions described herein. Theseinstructions, when executed by processor(s) 802 cause various operationsto implement the disclosed embodiments.

As used herein, the terms “machine-readable medium,” “computer-readablemedium” and “device-readable medium” mean the same thing and may be usedinterchangeably in this disclosure. The terms include a single medium ormultiple media (e.g., a centralized or distributed database, and/orassociated caches and servers) that store the one or more instructionsor data structures. The terms shall also be taken to include anytangible medium that is capable of storing, encoding or carryinginstructions for execution by the machine and that cause the machine toperform any one or more of the methodologies of the present invention,or that is capable of storing, encoding or carrying data structuresutilized by or associated with such instructions. The terms shallaccordingly be taken to include, but not be limited to, solid-statememories, and optical and magnetic media. Specific examples ofmachine-readable media, computer-readable media and/or device-readablemedia include non-volatile memory, including by way of examplesemiconductor memory devices, e.g., erasable programmable read-onlymemory (EPROM), electrically erasable programmable read-only memory(EEPROM), and flash memory devices; magnetic disks such as internal harddisks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The terms machine-readable media, computer-readable media, anddevice-readable media specifically exclude non-statutory signals per se,which are covered under the term “signal medium” discussed below.

Signal Medium

The term “signal medium” shall be taken to include any form of modulateddata signal and signals per se. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a matter as to encode information in the signal.

EXAMPLE EMBODIMENTS Example 1

A method for selecting and ranking service extensions, comprising:

receiving a query or an inference related to a user;

identifying any bots that activate in response to the query orinference;

ranking the activated bots by performing acts for each activated botcomprising:

identifying a set of features corresponding to the activated bot;

presenting at least a portion of the set of features to a ranking andresponse selection model trained via a machine learning process;

identifying a rank for the activated bot using the ranking and responseselection model and the portion of the set of features; and

placing the activated bot in rank order compared to other activated botsto create a ranked list of bots;

selecting at least one bot from the ranked list of bots; and

presenting an output of at least one bot to the user.

Example 2

The method of example 1, further comprising converting the query fromone format to another format for processing.

Example 3

The method of example 1, wherein the set of features comprises at leastone of:

features extracted from results of a language understanding model;

features extracted from how well a conversation progress under thehypothesis that a specific response is accepted and presented to theuser;

features extracted from statistics collected for models used by tasksactivities of the activated bot;

features extracted from the activated bot configuration; and

features extracted from an input source.

Example 4

The method of example 1, wherein the set of features further comprisesat least one of:

a correlation metric between a language understanding model of a bot anda built-in language understanding model;

a list of tasks that the bot handles; and

at least one metric describing at least one query that causes the bot totrigger.

Example 5

The method of example 1, 2, 3, or 4, further comprising:

retrieving a set of curated queries;

executing the set of curated queries against a submitted bot;

gathering a set of features characterizing the submitted bot response tothe set of curated queries; and

storing the set of features.

Example 6

The method of example 5, further comprising:

extracting from the submitted bot a set of features comprising taskshandled by the bot.

Example 7

The method of example 5, further comprising:

identifying a set of positive trigger examples or a set of negativetrigger examples or both based on the submitted bot response to the setof curated queries.

Example 8

The method of example 1, 2, 3 or 4, further comprising:

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and responseselection process from the set of represented bots and the training setof curated queries.

Example 9

A computing system comprising:

a processor and executable instructions accessible on a machine-readablemedium that, when executed, cause the system to perform operationscomprising:

receiving a query or an inference related to a user;

identifying any bots that activate in response to the query orinference;

ranking the activated bots by performing operations for each activatedbot comprising:

identifying a set of features corresponding to the activated bot;

presenting at least a portion of the set of features to a ranking andresponse selection model trained via a machine learning process;

identifying a rank for the activated bot using the ranking and responseselection model and the portion of the set of features; and

placing the activated bot in rank order compared to other activated botsto create a ranked list of bots;

selecting at least one bot from the ranked list of bots; and

presenting an output of the at least one bot to the user.

Example 10

The system of example 9, wherein the set of features comprises featurescorresponding to the activated bot in the context of other activatedbots and features calculated from a set of curated queries.

Example 11

The system of example 9, wherein the set of features comprises at leastone of:

a confidence score of a language understanding model;

an indicator of entities detected by the language understanding model;

a confidence score distribution;

a frequency of triggering on top of an existing intent; and

an indicator of the type of queries that cause triggering.

Example 12

The system of example 9, wherein the set of features further comprisesat least one of:

a correlation metric between a language understanding model of a bot anda built-in language understanding model;

a list of tasks that the bot handles; and

at least one metric describing at least one query that causes the bot totrigger.

Example 13

The system of example 9, 10, 11 or 12, further comprising:

retrieving a set of curated queries;

executing the set of curated queries against a submitted bot;

gathering a set of features characterizing the submitted bot response tothe set of curated queries; and

storing the set of features.

Example 14

The system of example 13, further comprising:

extracting from the submitted bot a set of features comprising taskshandled by the bot.

Example 15

The system of example 13, further comprising:

identifying a set of positive trigger examples or a set of negativetrigger examples or both based on the submitted bot response to the setof curated queries.

Example 16

The system of example 9, 10, 11 or 12, further comprising:

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and responseselection process from the set of represented bots and the training setof curated queries.

Example 17

A machine-readable medium having executable instructions encodedthereon, which, when executed by at least one processor of a machine,cause the machine to perform operations comprising:

train a ranking and response selection model by performing operationscomprising:

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and responseselection process from the set of represented bots and the training setof curated queries;

receive a query from a user or an inference related to a user;

identify any bots that activate in response to the query;

rank the activated bots by performing operations for each activated botcomprising:

identify a set of features corresponding to the activated bot in thecontext of other activated bots;

identify a second set of features calculated from a second set ofcurated queries;

combine the set of features and the second set of features into a set ofextracted features;

present a subset of the set of extracted features to the ranking andresponse selection model;

identify a rank for the activated bot using the ranking and responseselection model and the subset of extracted features; and

place the activated bot in rank order compared to other activated botsto create a ranked list of bots;

select at least one bot from the ranked list of bots; and

present an output of the at least one bot to the user.

Example 18

The machine-readable medium of example 17, wherein the machine learningprocess comprises at least one of:

a regression based method;

an instance based method;

a regularization method;

a decision tree method;

a Bayesian method;

a clustering method;

an association rule learning method;

a neural network method;

a deep learning method;

a dimensionality reduction method; and

an ensemble method.

Example 19

The machine-readable medium of example 17, wherein the second set offeatures is extracted by operations comprising:

retrieve a set of curated queries;

execute the set of curated queries against a submitted bot;

gather the second set of features characterizing the submitted botresponse to the set of curated queries; and

store the set of features.

Example 20

The machine-readable medium of example 17, 18, 19 or 20 wherein the setof features or second set of features comprises at least one of:

features extracted from results of a language understanding model;

features extracted from how well a conversation progress under thehypothesis that a specific response is accepted and presented to theuser;

features extracted from statistics collected for models used by tasksactivities of the activated bot;

features extracted from the activated bot configuration; and

features extracted from an input source.

Example 21

A method for selecting and ranking service extensions, comprising:

receiving a query or an inference related to a user;

identifying any bots that activate in response to the query orinference;

ranking the activated bots by performing acts for each activated botcomprising:

identifying a set of features corresponding to the activated bot;

presenting at least a portion of the set of features to a ranking andresponse selection model trained via a machine learning process;

identifying a rank for the activated bot using the ranking and responseselection model and the portion of the set of features; and

placing the activated bot in rank order compared to other activated botsto create a ranked list of bots;

selecting at least one bot from the ranked list of bots; and

presenting an output of at least one bot to the user.

Example 22

The method of example 21, further comprising converting the query fromone format to another format for processing.

Example 23

The method of example 21, or 22, wherein the set of features comprisesat least one of:

features extracted from results of a language understanding model;

features extracted from how well a conversation progress under thehypothesis that a specific response is accepted and presented to theuser;

features extracted from statistics collected for models used by tasksactivities of the activated bot;

features extracted from the activated bot configuration; and

features extracted from an input source.

Example 24

The method of example 21, 22, or 23, wherein the set of features furthercomprises at least one of:

a correlation metric between a language understanding model of a bot anda built-in language understanding model;

a list of tasks that the bot handles; and

at least one metric describing at least one query that causes the bot totrigger.

Example 25

The method of example 21, 22, 23, or 24, further comprising:

retrieving a set of curated queries;

executing the set of curated queries against a submitted bot;

gathering a set of features characterizing the submitted bot response tothe set of curated queries; and

storing the set of features.

Example 26

The method of example 25, further comprising:

extracting from the submitted bot a set of features comprising taskshandled by the bot.

Example 27

The method of example 25, further comprising:

identifying a set of positive trigger examples or a set of negativetrigger examples or both based on the submitted bot response to the setof curated queries.

Example 28

The method of example 21, 22, 23, 24, 25, 26 or 27, further comprising:

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and responseselection process from the set of represented bots and the training setof curated queries.

Example 29

The method of example 21, 22, 23, 24, 25, 26, 27 or 28, wherein the setof features comprises features corresponding to the activated bot in thecontext of other activated bots and features calculated from a set ofcurated queries.

Example 30

The method of example 21, 22, 23, 24, 25, 26, 27, 28, or 29, wherein theset of features comprises at least one of:

a confidence score of a language understanding model;

an indicator of entities detected by the language understanding model;

a confidence score distribution;

a frequency of triggering on top of an existing intent; and

an indicator of the type of queries that cause triggering.

Example 31

The method of example 21, 22, 23, 24, 25, 26, 27, 28, 29, or 30, whereinthe machine learning process comprises at least one of:

a regression based method;

an instance based method;

a regularization method;

a decision tree method;

a Bayesian method;

a clustering method;

an association rule learning method;

a neural network method;

a deep learning method;

a dimensionality reduction method; and

an ensemble method.

Example 32

The method of example 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 or 31,wherein identifying the set of features corresponding to the activatedbot comprises:

identify a first set of features corresponding to the activated bot inthe context of other activated bots;

identify a second set of features calculated from a set of curatedqueries;

combine the set of features and the second set of features into the setof features corresponding to the activated bot;

Example 33

The method of example 32, wherein identifying a second set of featurescalculated from a set of curated queries comprises identifying a closestset of features from among a set of features.

Example 34

An apparatus comprising means to perform a method as in any precedingexample.

Example 35

Machine-readable storage including machine-readable instructions, whenexecuted, to implement a method or realize an apparatus as in anypreceding example.

CONCLUSION

In view of the many possible embodiments to which the principles of thepresent invention and the forgoing examples may be applied, it should berecognized that the examples described herein are meant to beillustrative only and should not be taken as limiting the scope of thepresent invention. Therefore, the invention as described hereincontemplates all such embodiments as may come within the scope of thefollowing claims and any equivalents thereto.

What is claimed is:
 1. A method for selecting and ranking service extensions, comprising: receiving a query or an inference related to a user; identifying any bots that activate in response to the query or inference; ranking the activated bots by performing acts for each activated bot comprising: identifying a set of features corresponding to the activated bot; presenting at least a portion of the set of features to a ranking and response selection model trained via a machine learning process; identifying a rank for the activated bot using the ranking and response selection model and the portion of the set of features; and placing the activated bot in rank order compared to other activated bots to create a ranked list of bots; selecting at least one bot from the ranked list of bots; and presenting an output of at least one bot to the user.
 2. The method of claim 1, further comprising converting the query from one format to another format for processing.
 3. The method of claim 1, wherein the set of features comprises at least one of: features extracted from results of a language understanding model; features extracted from how well a conversation progress under the hypothesis that a specific response is accepted and presented to the user; features extracted from statistics collected for models used by tasks activities of the activated bot; features extracted from the activated bot configuration; and features extracted from an input source.
 4. The method of claim 1, wherein the set of features further comprises at least one of: a correlation metric between a language understanding model of a bot and a built-in language understanding model; a list of tasks that the bot handles; and at least one metric describing at least one query that causes the bot to trigger.
 5. The method of claim 1, further comprising: retrieving a set of curated queries; executing the set of curated queries against a submitted bot; gathering a set of features characterizing the submitted bot response to the set of curated queries; and storing the set of features.
 6. The method of claim 5, further comprising: extracting from the submitted bot a set of features comprising tasks handled by the bot.
 7. The method of claim 5, further comprising: identifying a set of positive trigger examples or a set of negative trigger examples or both based on the submitted bot response to the set of curated queries.
 8. The method of claim 1, further comprising: identifying a set of representative bots; identifying a training set of curated queries; and using a machine learning process, creating the ranking and response selection process from the set of represented bots and the training set of curated queries.
 9. A computing system comprising: a processor and executable instructions accessible on a machine-readable medium that, when executed, cause the system to perform operations comprising: receiving a query or an inference related to a user; identifying any bots that activate in response to the query or inference; ranking the activated bots by performing operations for each activated bot comprising: identifying a set of features corresponding to the activated bot; presenting at least a portion of the set of features to a ranking and response selection model trained via a machine learning process; identifying a rank for the activated bot using the ranking and response selection model and the portion of the set of features; and placing the activated bot in rank order compared to other activated bots to create a ranked list of bots; selecting at least one bot from the ranked list of bots; and presenting an output of the at least one bot to the user.
 10. The system of claim 9, wherein the set of features comprises features corresponding to the activated bot in the context of other activated bots and features calculated from a set of curated queries.
 11. The system of claim 9, wherein the set of features comprises at least one of: a confidence score of a language understanding model; an indicator of entities detected by the language understanding model; a confidence score distribution; a frequency of triggering on top of an existing intent; and an indicator of the type of queries that cause triggering.
 12. The system of claim 9, wherein the set of features further comprises at least one of: a correlation metric between a language understanding model of a bot and a built-in language understanding model; a list of tasks that the bot handles; and at least one metric describing at least one query that causes the bot to trigger.
 13. The system of claim 9, further comprising: retrieving a set of curated queries; executing the set of curated queries against a submitted bot; gathering a set of features characterizing the submitted bot response to the set of curated queries; and storing the set of features.
 14. The system of claim 13, further comprising: extracting from the submitted bot a set of features comprising tasks handled by the bot.
 15. The system of claim 13, further comprising: identifying a set of positive trigger examples or a set of negative trigger examples or both based on the submitted bot response to the set of curated queries.
 16. The system of claim 9, further comprising: identifying a set of representative bots; identifying a training set of curated queries; and using a machine learning process, creating the ranking and response selection process from the set of represented bots and the training set of curated queries.
 17. A machine-readable medium having executable instructions encoded thereon, which, when executed by at least one processor of a machine, cause the machine to perform operations comprising: train a ranking and response selection model by performing operations comprising: identifying a set of representative bots; identifying a training set of curated queries; and using a machine learning process, creating the ranking and response selection process from the set of represented bots and the training set of curated queries; receive a query from a user or an inference related to a user; identify any bots that activate in response to the query; rank the activated bots by performing operations for each activated bot comprising: identify a set of features corresponding to the activated bot in the context of other activated bots; identify a second set of features calculated from a second set of curated queries; combine the set of features and the second set of features into a set of extracted features; present a subset of the set of extracted features to the ranking and response selection model; identify a rank for the activated bot using the ranking and response selection model and the subset of extracted features; and place the activated bot in rank order compared to other activated bots to create a ranked list of bots; select at least one bot from the ranked list of bots; and present an output of the at least one bot to the user.
 18. The machine-readable medium of claim 17, wherein the machine learning process comprises at least one of: a regression based method; an instance based method; a regularization method; a decision tree method; a Bayesian method; a clustering method; an association rule learning method; a neural network method; a deep learning method; a dimensionality reduction method; and an ensemble method.
 19. The machine-readable medium of claim 17, wherein the second set of features is extracted by operations comprising: retrieve a set of curated queries; execute the set of curated queries against a submitted bot; gather the second set of features characterizing the submitted bot response to the set of curated queries; and store the set of features.
 20. The machine-readable medium of claim 17, wherein the set of features or second set of features comprises at least one of: features extracted from results of a language understanding model; features extracted from how well a conversation progress under the hypothesis that a specific response is accepted and presented to the user; features extracted from statistics collected for models used by tasks activities of the activated bot; features extracted from the activated bot configuration; and features extracted from an input source. 